bgpd: RD caching by soumyar-roy · Pull Request #21266 · FRRouting/frr

soumyar-roy · 2026-03-19T23:38:01Z

RD Caching - Summary
Problem: FRR auto-derives EVPN Route Distinguishers (RDs) as Type 1 format (RouterID:2-byte-number). Since VNIs are 24-bit (up to 16M) but the 2-byte field only holds up to 65535, FRR can't embed the VNI directly. Instead it uses a sequential index from bf_assign_index() which is non-deterministic across restarts. This means after a bgpd restart, VRFs and VNIs get different RD values, causing EVPN peers to see them as entirely new routes rather than updates to existing ones — leading to traffic disruption during warm/graceful restart.
Solution: Persist the rd_id assignments to disk so they survive bgpd restarts.
How it works:
State files (/var/lib/frr/.bgp_vrf_rd.txt and .bgp_vni_rd.txt) store the mapping of VRF name → rd_id and VNI → rd_id.
On startup, the state files are loaded into in-memory hash tables. The corresponding bits in the rd_idspace bitfield are reserved so new allocations don't collide.
When a VRF/VNI is configured, it looks up the hash first — if a cached rd_id exists, it reuses it. Otherwise it allocates a new one and appends to the state file.
When a VRF/VNI is deleted, its entry is removed from the hash and the state file is rewritten from the hash (atomic write via tmp + rename).
Orphan cleanup: After config is fully loaded, a bgp_config_end hook walks both hashes and removes entries that were never claimed (VRFs/VNIs that were in the state file but no longer in the running config). Their rd_id bits are freed and state files are rewritten.
Result: RDs remain stable across restarts, enabling seamless EVPN graceful restart without traffic loss

# Design Proposal: Persistent Route Distinguisher Index for EVPN Graceful Restart

Problem Statement

FRR's BGP daemon auto-derives Route Distinguishers (RDs) for EVPN VRFs and
L2VNIs using Type 1 format: RouterID:rd_id, where rd_id is a 16-bit
sequential index allocated from a shared bitfield (bm->rd_idspace) at
creation time (see form_auto_rd() in bgp_rd.c).

This index is non-deterministic. It depends on the order in which VRFs and
VNIs happen to be created during startup, which in turn depends on config
parse order, zebra notification timing, and kernel state. After a daemon
restart, a VNI that previously had RD 10.0.0.1:2 may now receive
10.0.0.1:5.

During EVPN Graceful Restart, the restarting speaker must re-advertise
routes with the same RD they carried before the restart. When the RD changes:

Peers treat the new RD routes as entirely new prefixes rather than
refreshed versions of existing ones.
Stale routes under the old RD are eventually purged, causing a transient
blackhole.

This is observable in production as brief traffic loss on every bgpd restart,
defeating the purpose of graceful restart.

Why Alternative Approaches Fall Short

"Configure explicit RDs via orchestration"

Operators can already set rd X:Y per VNI and per VRF. However:

Auto-derivation exists precisely because manual RD management may not
scale. A typical leaf switch may host 500+ L2VNIs plus several L3VNIs;
maintaining unique, collision-free RD assignments across every node in a
provisioning system is significant operational overhead.
Exiting customer usages are based upon auto RD

"Use a different RD format"

RFC 4364 defines three RD types:

Type	Format	Discriminator field
0	ASN(2):Value(4)	32-bit
1	IP(4):Value(2)	16-bit
2	ASN4(4):Value(2)	16-bit

FRR uses Type 1 (RouterID:index). Switching to Type 0 would give a 32-bit
discriminator, but:

VNIs are 24-bit, so using VNI directly as the discriminator would work
for L2VNIs but not for L3VNIs (VRFs), where no natural numeric key
exists.
Type 0 uses ASN in the high field, so all VRFs/VNIs sharing the same ASN
in different routers would need globally coordinated discriminator values
-- the same orchestration problem.
Type 2 has the same 16-bit limitation as Type 1.

No format change eliminates the need for a stable mapping between the
VRF/VNI identity and its assigned index. The index must be remembered
across restarts.

Proposed Design

Overview

Persist the (VRF-name -> rd_id) and (VNI -> rd_id) mappings to
lightweight state files under frr_libstatedir. On startup, reload these
mappings before config is processed so that every VRF/VNI reclaims its
previous rd_id.

Scope of State

The state is minimal. Each file stores a mapping from an identity (VRF name
or VNI number) to the 16-bit rd_id index that form_auto_rd() uses to
build the RD as RouterID:rd_id.

VRF file ($statedir/.bgp_vrf_rd.txt): one line per VRF.
Format: <vrf-name> <rd_id>, e.g. default 1, vrf-blue 3.
The VRF name identifies the BGP instance; the rd_id is the index used
in the auto-derived RD (10.0.0.1:1, 10.0.0.1:3).
Typical size: tens of bytes.
VNI file ($statedir/.bgp_vni_rd.txt): one line per L2VNI.
Format: <vni> <rd_id>, e.g. 100 2, 200 4.
The VNI number identifies the EVPN instance; the rd_id is the index
used in the auto-derived RD (10.0.0.1:2, 10.0.0.1:4).
Typical size: a few KB even at 4K VNIs.

The files contain only the mapping between a name/VNI and its 16-bit index.
No routing state, no adjacency information, no timers. If the files are
missing or corrupt, bgpd falls back to the current behavior (allocate fresh
indices) with no other side-effects.

Lifecycle

 bgpd start
     |
     v
 bgp_init()
     |-- bgp_load_vrf_rd_statefile()   # parse file -> hash table, reserve bits
     |-- bgp_load_vni_rd_statefile()   # parse file -> hash table, reserve bits
     v
 config is loaded (frr.conf / vtysh)
     |-- VRF created -> bgp_assign_or_restore_vrf_rd_id()
     |       find in hash? -> restore rd_id        (cache hit)
     |       not found?    -> bf_assign_index()    (new allocation, append to file)
     |
     |-- VNI created (from zebra / config) -> bgp_evpn_assign_rd_id_for_vni()
     |       find in hash? -> restore rd_id        (cache hit)
     |       not found?    -> bf_assign_index()    (new allocation, append to file)
     v
 bgp_config_end hook
     |-- bgp_rd_state_cleanup_after_config()
     |       for each hash entry not marked 'used':
     |           release rd_id bit, remove from hash, rewrite file
     v
 normal operation
     |-- VRF/VNI deleted at runtime -> remove from hash, rewrite file
     v
 bgpd shutdown (SIGTERM)
     |-- bm->terminating = true
     |-- bgp_free() per instance: removes from hash, skips file rewrite
     |-- bgp_exit(): frees hash memory
     |-- state files on disk: UNCHANGED (preserved for next startup)

Data Structures

struct vrf_rd_state_entry {
    char     *name;      /* VRF name ("default", "vrf-blue", ...) */
    uint16_t  rd_id;     /* index used in form_auto_rd() */
    bool      used;      /* marked true when VRF is created from config */
    struct vrf_rd_state_hash_item item;  /* typesafe hash link */
};

struct vni_rd_state_entry {
    uint32_t  vni;       /* L2VNI number */
    uint16_t  rd_id;
    bool      used;
    struct vni_rd_state_hash_item item;
};

Hash tables use DECLARE_HASH (FRR's typesafe hash) keyed by VRF name
and VNI respectively. They live in struct bgp_master alongside the
existing rd_idspace bitfield.

File Format

Plain text, one entry per line:

# .bgp_vrf_rd.txt
default 1
vrf-blue 3

# .bgp_vni_rd.txt
100 2
200 4

Atomic writes use tmp-file + rename() to avoid corruption on crash.
Append-only writes are used for single-entry additions during normal
operation to avoid O(n) rewrites.

Failure Modes

Scenario	Behavior
State file missing	Fresh index allocation (existing behavior)
State file has corrupt/malformed lines	Skip line with `zlog_warn`, process rest
State file has rd_id > UINT16_MAX	Skip line with `zlog_warn`
Duplicate entries in file	First entry wins, duplicates freed
State dir not writable	Warn, continue without persistence
State file deleted between restarts	Fresh allocation, no crash

Orphan Cleanup

When a VRF or VNI is removed from the configuration between restarts, its
entry in the state file becomes orphaned. The bgp_config_end hook runs
bgp_rd_state_cleanup_after_config() which:

Iterates all hash entries.
Any entry not marked used (i.e., no VRF/VNI claimed it during config
load) is removed from the hash and its rd_id bit is released.
The state file is rewritten atomically.

This ensures the state files do not grow unboundedly over time.

Files Modified

File	Changes
`bgpd/bgpd.h`	State entry structs, `PREDECL_HASH`, `DECLARE_HASH`, hash head fields in `bgp_master`
`bgpd/bgpd.c`	Load/save/rewrite state files, assign-or-restore functions, orphan cleanup hook, integration into `bgp_free()`
`bgpd/bgp_evpn.c`	VNI-side assign-or-restore, log-to-statefile, rewrite, cleanup in `bgp_evpn_free()`
`bgpd/bgp_main.c`	Free hash entries at shutdown

What This Does NOT Change

No new CLI commands or configuration knobs.
No changes to BGP protocol behavior, packet formats, or FSM.
No changes to how RDs are derived (form_auto_rd() is untouched).
No changes to graceful restart timer handling or stale route processing.
Manually configured RDs (rd X:Y) are unaffected; the state file is
only consulted for auto-derived RDs.
If the state file is absent, behavior is identical to current FRR
(fresh sequential allocation).

Testing

Unit: State file parse/write with corrupt data, duplicate entries,
missing files.
Topotest (bgp_evpn_gr):
- Verify state files are created with correct entries after startup.
- Kill and restart bgpd; verify RD values are identical before and after.
- Remove a VRF from config, restart; verify orphan entry is cleaned from
  state file.

UT log>>>

Part1: Trigger GR and see, RD doesn’t trigger across reboot

VXLAN flooding: Enabled
Number of L2 VNIs: 4
Number of L3 VNIs: 2
Flags: * - Kernel
VNI Type RD Import RT Export RT MAC-VRF Site-of-Origin Tenant VRF

1000114 L2 6.0.0.1:3 4640:1000114 4640:1000114 vrf1
1000112 L2 6.0.0.1:4 4640:1000112 4640:1000112 vrf1
1000113 L2 6.0.0.1:5 4640:1000113 4640:1000113 vrf2
1000111 L2 6.0.0.1:6 4640:1000111 4640:1000111 vrf2
104001 L3 144.1.1.2:2 4640:104001 4640:104001 vrf1
104002 L3 144.1.1.6:9 4640:104002 4640:104002 vrf2
bordertor-11# exit
root@bordertor-11:mgmt:/var/home/cumulus# sudo kill -9 pidof zebra
sudo kill -9 pidof bgpd
sudo systemctl stop frr
root@bordertor-11:mgmt:/var/home/cumulus# sudo sed -i -e 's/cumulus_mlag/cumulus_mlag -K 60/g' /etc/frr/daemons
sudo systemctl start frr
sudo sed -i -e 's/cumulus_mlag -K 60/cumulus_mlag/g' /etc/frr/daemons
root@bordertor-11:mgmt:/var/home/cumulus# vtysh

bordertor-11# show bgp l2vpn evpn vni
Advertise Gateway Macip: Disabled
Advertise SVI Macip: Disabled
Advertise All VNI flag: Enabled
BUM flooding: Head-end replication
VXLAN flooding: Enabled
Number of L2 VNIs: 4
Number of L3 VNIs: 2
Flags: * - Kernel
VNI Type RD Import RT Export RT MAC-VRF Site-of-Origin Tenant VRF

1000114 L2 6.0.0.1:3 4640:1000114 4640:1000114 vrf1
1000112 L2 6.0.0.1:4 4640:1000112 4640:1000112 vrf1
1000113 L2 6.0.0.1:5 4640:1000113 4640:1000113 vrf2
1000111 L2 6.0.0.1:6 4640:1000111 4640:1000111 vrf2
104001 L3 144.1.1.2:2 4640:104001 4640:104001 vrf1
104002 L3 144.1.1.6:9 4640:104002 4640:104002 vrf2
bordertor-11# exit
root@bordertor-11:mgmt:/var/home/cumulus# sudo kill -9 pidof zebra
sudo kill -9 pidof bgpd
sudo systemctl stop frr
root@bordertor-11:mgmt:/var/home/cumulus# sudo sed -i -e 's/cumulus_mlag/cumulus_mlag -K 60/g' /etc/frr/daemons
sudo systemctl start frr
sudo sed -i -e 's/cumulus_mlag -K 60/cumulus_mlag/g' /etc/frr/daemons
root@bordertor-11:mgmt:/var/home/cumulus# vtysh

bordertor-11# show bgp l2vpn evpn vni
Advertise Gateway Macip: Disabled
Advertise SVI Macip: Disabled
Advertise All VNI flag: Enabled
BUM flooding: Head-end replication
VXLAN flooding: Enabled
Number of L2 VNIs: 4
Number of L3 VNIs: 2
Flags: * - Kernel
VNI Type RD Import RT Export RT MAC-VRF Site-of-Origin Tenant VRF

1000114 L2 6.0.0.1:3 4640:1000114 4640:1000114 vrf1
1000112 L2 6.0.0.1:4 4640:1000112 4640:1000112 vrf1
1000113 L2 6.0.0.1:5 4640:1000113 4640:1000113 vrf2
1000111 L2 6.0.0.1:6 4640:1000111 4640:1000111 vrf2
104001 L3 144.1.1.2:2 4640:104001 4640:104001 vrf1
104002 L3 144.1.1.6:9 4640:104002 4640:104002 vrf2
bordertor-11# exit
root@bordertor-11:mgmt:/var/home/cumulus# sudo kill -9 pidof zebra
sudo kill -9 pidof bgpd
sudo systemctl stop frr
root@bordertor-11:mgmt:/var/home/cumulus# sudo sed -i -e 's/cumulus_mlag/cumulus_mlag -K 60/g' /etc/frr/daemons
sudo systemctl start frr
sudo sed -i -e 's/cumulus_mlag -K 60/cumulus_mlag/g' /etc/frr/daemons
root@bordertor-11:mgmt:/var/home/cumulus# vtysh

bordertor-11# show bgp l2vpn evpn vni
Advertise Gateway Macip: Disabled
Advertise SVI Macip: Disabled
Advertise All VNI flag: Enabled
BUM flooding: Head-end replication
VXLAN flooding: Enabled
Number of L2 VNIs: 4
Number of L3 VNIs: 2
Flags: * - Kernel
VNI Type RD Import RT Export RT MAC-VRF Site-of-Origin Tenant VRF

1000114 L2 6.0.0.1:3 4640:1000114 4640:1000114 vrf1
1000112 L2 6.0.0.1:4 4640:1000112 4640:1000112 vrf1
1000113 L2 6.0.0.1:5 4640:1000113 4640:1000113 vrf2
1000111 L2 6.0.0.1:6 4640:1000111 4640:1000111 vrf2
104001 L3 144.1.1.2:2 4640:104001 4640:104001 vrf1
104002 L3 144.1.1.6:9 4640:104002 4640:104002 vrf2
bordertor-11# exit
root@bordertor-11:mgmt:/var/home/cumulus# sudo kill -9 pidof zebra
sudo kill -9 pidof bgpd
sudo systemctl stop frr
root@bordertor-11:mgmt:/var/home/cumulus# sudo sed -i -e 's/cumulus_mlag/cumulus_mlag -K 60/g' /etc/frr/daemons
sudo systemctl start frr
sudo sed -i -e 's/cumulus_mlag -K 60/cumulus_mlag/g' /etc/frr/daemons
root@bordertor-11:mgmt:/var/home/cumulus# vtysh

bordertor-11# show bgp l2vpn evpn vni
Advertise Gateway Macip: Disabled
Advertise SVI Macip: Disabled
Advertise All VNI flag: Enabled
BUM flooding: Head-end replication
VXLAN flooding: Enabled
Number of L2 VNIs: 4
Number of L3 VNIs: 2
Flags: * - Kernel
VNI Type RD Import RT Export RT MAC-VRF Site-of-Origin Tenant VRF

1000114 L2 6.0.0.1:3 4640:1000114 4640:1000114 vrf1
1000112 L2 6.0.0.1:4 4640:1000112 4640:1000112 vrf1
1000113 L2 6.0.0.1:5 4640:1000113 4640:1000113 vrf2
1000111 L2 6.0.0.1:6 4640:1000111 4640:1000111 vrf2
104001 L3 144.1.1.2:2 4640:104001 4640:104001 vrf1
104002 L3 144.1.1.6:9 4640:104002 4640:104002 vrf2
bordertor-11# sh run

bordertor-11# exit
root@bordertor-11:mgmt:/var/home/cumulus# sudo kill -9 pidof zebra
sudo kill -9 pidof bgpd
sudo systemctl stop frr
root@bordertor-11:mgmt:/var/home/cumulus# sudo sed -i -e 's/cumulus_mlag/cumulus_mlag -K 60/g' /etc/frr/daemons
sudo systemctl start frr
sudo sed -i -e 's/cumulus_mlag -K 60/cumulus_mlag/g' /etc/frr/daemons
root@bordertor-11:mgmt:/var/home/cumulus# vtysh

bordertor-11# show bgp l2vpn evpn vni
Advertise Gateway Macip: Disabled
Advertise SVI Macip: Disabled
Advertise All VNI flag: Enabled
BUM flooding: Head-end replication
VXLAN flooding: Enabled
Number of L2 VNIs: 4
Number of L3 VNIs: 2
Flags: * - Kernel
VNI Type RD Import RT Export RT MAC-VRF Site-of-Origin Tenant VRF

1000114 L2 6.0.0.1:3 4640:1000114 4640:1000114 vrf1
1000112 L2 6.0.0.1:4 4640:1000112 4640:1000112 vrf1
1000113 L2 6.0.0.1:5 4640:1000113 4640:1000113 vrf2
1000111 L2 6.0.0.1:6 4640:1000111 4640:1000111 vrf2
104001 L3 144.1.1.2:2 4640:104001 4640:104001 vrf1
104002 L3 144.1.1.6:9 4640:104002 4640:104002 vrf2
bordertor-11# exit
root@bordertor-11:mgmt:/var/home/cumulus# sudo kill -9 pidof zebra
sudo kill -9 pidof bgpd
sudo systemctl stop frr
root@bordertor-11:mgmt:/var/home/cumulus# sudo sed -i -e 's/cumulus_mlag/cumulus_mlag -K 60/g' /etc/frr/daemons
sudo systemctl start frr
sudo sed -i -e 's/cumulus_mlag -K 60/cumulus_mlag/g' /etc/frr/daemons
root@bordertor-11:mgmt:/var/home/cumulus# vtysh

bordertor-11# show bgp l2vpn evpn vni
Advertise Gateway Macip: Disabled
Advertise SVI Macip: Disabled
Advertise All VNI flag: Enabled
BUM flooding: Head-end replication
VXLAN flooding: Enabled
Number of L2 VNIs: 4
Number of L3 VNIs: 2
Flags: * - Kernel
VNI Type RD Import RT Export RT MAC-VRF Site-of-Origin Tenant VRF

1000114 L2 6.0.0.1:3 4640:1000114 4640:1000114 vrf1
1000112 L2 6.0.0.1:4 4640:1000112 4640:1000112 vrf1
1000113 L2 6.0.0.1:5 4640:1000113 4640:1000113 vrf2
1000111 L2 6.0.0.1:6 4640:1000111 4640:1000111 vrf2
104001 L3 144.1.1.2:2 4640:104001 4640:104001 vrf1
104002 L3 144.1.1.6:9 4640:104002 4640:104002 vrf2
bordertor-11# exit
root@bordertor-11:mgmt:/var/home/cumulus# sudo sed -i -e 's/cumulus_mlag/cumulus_mlag -K 60/g' /etc/frr/daemons
sudo systemctl start frr
sudo sed -i -e 's/cumulus_mlag -K 60/cumulus_mlag/g' /etc/frr/daemons
root@bordertor-11:mgmt:/var/home/cumulus# sudo sed -i -e 's/cumulus_mlag/cumulus_mlag -K 60/g' /etc/frr/daemons
sudo systemctl start frr
sudo sed -i -e 's/cumulus_mlag -K 60/cumulus_mlag/g' /etc/frr/daemons
root@bordertor-11:mgmt:/var/home/cumulus# vtysh

bordertor-11# show bgp l2vpn evpn vni
Advertise Gateway Macip: Disabled
Advertise SVI Macip: Disabled
Advertise All VNI flag: Enabled
BUM flooding: Head-end replication
VXLAN flooding: Enabled
Number of L2 VNIs: 4
Number of L3 VNIs: 2
Flags: * - Kernel
VNI Type RD Import RT Export RT MAC-VRF Site-of-Origin Tenant VRF

1000114 L2 6.0.0.1:3 4640:1000114 4640:1000114 vrf1
1000112 L2 6.0.0.1:4 4640:1000112 4640:1000112 vrf1
1000113 L2 6.0.0.1:5 4640:1000113 4640:1000113 vrf2
1000111 L2 6.0.0.1:6 4640:1000111 4640:1000111 vrf2
104001 L3 144.1.1.2:2 4640:104001 4640:104001 vrf1
104002 L3 144.1.1.6:9 4640:104002 4640:104002 vrf2
bordertor-11#

Part 2>>>>>>>>>>>>>>>>>>Config based trigger>>>>>>>
bordertor-11# sh run
Building configuration...

Current configuration:
!
frr version 10.0.3
frr defaults datacenter
hostname bordertor-11
log file /var/log/frr/bgpd.log
log syslog informational
log timestamp precision 6
service integrated-vtysh-config
!
ip prefix-list LOCAL_HOST_VRF1 seq 1 permit 50.1.110.0/24
ip prefix-list LOCAL_HOST_VRF1 seq 2 deny any
ip prefix-list LOCAL_HOST_VRF2 seq 1 permit 60.1.110.0/24
ip prefix-list LOCAL_HOST_VRF2 seq 2 deny any
!
ipv6 prefix-list LOCAL_HOST_VRF1_v6 seq 1 permit 2050:1:1:110::/64
ipv6 prefix-list LOCAL_HOST_VRF1_v6 seq 2 deny any
ipv6 prefix-list LOCAL_HOST_VRF2_v6 seq 1 permit 2060:1:1:110::/64
ipv6 prefix-list LOCAL_HOST_VRF2_v6 seq 2 deny any
!
vrf mgmt
exit-vrf
!
vrf vrf1
vni 104001
exit-vrf
!
vrf vrf2
vni 104002
exit-vrf
!
router bgp 660000
bgp router-id 6.0.0.1
bgp graceful-restart
bgp bestpath as-path multipath-relax
bgp bestpath compare-routerid
neighbor TOR_LEAF_SPINE peer-group
neighbor TOR_LEAF_SPINE advertisement-interval 0
neighbor TOR_LEAF_SPINE timers 3 10
neighbor TOR_LEAF_SPINE timers connect 5
neighbor TOR_LEAF_SPINE capability extended-nexthop
neighbor swp1 interface peer-group TOR_LEAF_SPINE
neighbor swp1 remote-as external
neighbor swp1 advertisement-interval 0
neighbor swp1 timers 3 10
neighbor swp1 timers connect 5
neighbor swp2 interface peer-group TOR_LEAF_SPINE
neighbor swp2 remote-as external
neighbor swp2 advertisement-interval 0
neighbor swp2 timers 3 10
neighbor swp2 timers connect 5
!
address-family ipv4 unicast
redistribute connected route-map ALLOW_LOBR
neighbor TOR_LEAF_SPINE allowas-in
neighbor swp1 allowas-in
neighbor swp2 allowas-in
maximum-paths 16
maximum-paths ibgp 64
exit-address-family
!
address-family l2vpn evpn
neighbor TOR_LEAF_SPINE activate
advertise-all-vni
exit-address-family
exit
!
router bgp 660000 vrf vrf1
bgp router-id 144.1.1.2
no bgp network import-check
neighbor 144.1.1.1 remote-as external
neighbor 144.1.1.1 advertisement-interval 0
neighbor 144.1.1.1 timers 3 9
neighbor 144.1.1.1 timers connect 10
neighbor 155.1.1.1 remote-as external
neighbor 155.1.1.1 advertisement-interval 0
neighbor 155.1.1.1 timers 3 9
neighbor 155.1.1.1 timers connect 10
neighbor 2144:1:1:1::1 remote-as external
neighbor 2144:1:1:1::1 advertisement-interval 0
neighbor 2144:1:1:1::1 timers 3 9
neighbor 2144:1:1:1::1 timers connect 10
neighbor 2155:1:1:1::1 remote-as external
neighbor 2155:1:1:1::1 advertisement-interval 0
neighbor 2155:1:1:1::1 timers 3 9
neighbor 2155:1:1:1::1 timers connect 10
!
address-family ipv4 unicast
network 50.1.1.112/32
redistribute connected route-map HOST_ALLOW_1
maximum-paths 64
maximum-paths ibgp 64
exit-address-family
!
address-family ipv6 unicast
network 2050:1:1:1::112/128
redistribute connected route-map HOST_ALLOW_1_v6
neighbor 2144:1:1:1::1 activate
neighbor 2155:1:1:1::1 activate
maximum-paths 64
maximum-paths ibgp 64
exit-address-family
!
address-family l2vpn evpn
advertise ipv4 unicast
advertise ipv6 unicast
exit-address-family
exit
!
router bgp 660000 vrf vrf2
bgp router-id 144.1.1.6
no bgp network import-check
neighbor 144.1.1.5 remote-as external
neighbor 144.1.1.5 advertisement-interval 0
neighbor 144.1.1.5 timers 3 9
neighbor 144.1.1.5 timers connect 10
neighbor 155.1.1.5 remote-as external
neighbor 155.1.1.5 advertisement-interval 0
neighbor 155.1.1.5 timers 3 9
neighbor 155.1.1.5 timers connect 10
neighbor 2144:1:1:2::5 remote-as external
neighbor 2144:1:1:2::5 advertisement-interval 0
neighbor 2144:1:1:2::5 timers 3 9
neighbor 2144:1:1:2::5 timers connect 10
neighbor 2155:1:1:2::5 remote-as external
neighbor 2155:1:1:2::5 advertisement-interval 0
neighbor 2155:1:1:2::5 timers 3 9
neighbor 2155:1:1:2::5 timers connect 10
!
address-family ipv4 unicast
network 60.1.1.111/32
redistribute connected route-map HOST_ALLOW_2
maximum-paths 64
maximum-paths ibgp 64
exit-address-family
!
address-family ipv6 unicast
network 2060:1:1:1::111/128
redistribute connected route-map HOST_ALLOW_2_v6
neighbor 2144:1:1:2::5 activate
neighbor 2155:1:1:2::5 activate
maximum-paths 64
maximum-paths ibgp 64
exit-address-family
!
address-family l2vpn evpn
advertise ipv4 unicast
advertise ipv6 unicast
exit-address-family
exit
!
route-map ALLOW_LO permit 10
match interface lo
!
route-map ALLOW_LOBR permit 10
match interface lo
!
route-map ALLOW_LOBR permit 20
match interface br_l3vni
!
route-map HOST_ALLOW_1 permit 1
match ip address prefix-list LOCAL_HOST_VRF1
!
route-map HOST_ALLOW_1_v6 permit 1
match ipv6 address prefix-list LOCAL_HOST_VRF1_v6
!
route-map HOST_ALLOW_2 permit 1
match ip address prefix-list LOCAL_HOST_VRF2
!
route-map HOST_ALLOW_2_v6 permit 1
match ipv6 address prefix-list LOCAL_HOST_VRF2_v6
!
end

root@bordertor-11:mgmt:/var/home/cumulus# cat /var/lib/frr/.bgp_vrf_rd.txt
default 1
vrf1 2
vrf2 9
root@bordertor-11:mgmt:/var/home/cumulus# cat /var/lib/frr/.bgp_vni_rd.txt
1000114 3
1000112 4
1000113 5
1000111 6
root@bordertor-11:mgmt:/var/home/cumulus# vtysh

bordertor-11# show bgp l2vpn evpn vni
Advertise Gateway Macip: Disabled
Advertise SVI Macip: Disabled
Advertise All VNI flag: Enabled
BUM flooding: Head-end replication
VXLAN flooding: Enabled
Number of L2 VNIs: 4
Number of L3 VNIs: 2
Flags: * - Kernel
VNI Type RD Import RT Export RT MAC-VRF Site-of-Origin Tenant VRF

1000114 L2 6.0.0.1:3 4640:1000114 4640:1000114 vrf1
1000112 L2 6.0.0.1:4 4640:1000112 4640:1000112 vrf1
1000113 L2 6.0.0.1:5 4640:1000113 4640:1000113 vrf2
1000111 L2 6.0.0.1:6 4640:1000111 4640:1000111 vrf2
104001 L3 144.1.1.2:2 4640:104001 4640:104001 vrf1
104002 L3 144.1.1.6:9 4640:104002 4640:104002 vrf2
--- RESULT 3b: PASS --- (same as TEST 2)
104001 L3 with vrf1 rd_id=2, orphan cleaned, all L2 VNIs unchanged
================================================================================

================================================================================
TEST Full VRF removal — stop FRR, remove BOTH the VRF definition block
("vrf vrf1 / vni 104001 / exit-vrf") AND "router bgp 660000 vrf vrf1"
from frr.conf. Start FRR. Then add both back, restart FRR.
Expect: vrf1 BGP instance is gone so vrf1 entry is removed from VRF
state file. 104001 demotes L3->L2 with rd_id=7.
On re-add, vrf1 gets rd_id=2 (lowest free bit).

--- Step 4a: Remove VRF block + BGP VRF instance from config, stop/start FRR ---
bordertor-11# exit
root@bordertor-11:mgmt:/var/home/cumulus# vi /etc/frr/frr.conf<<<<<remove vrf vrf1 from global and BGP
root@bordertor-11:mgmt:/var/home/cumulus# systemctl stop frr
root@bordertor-11:mgmt:/var/home/cumulus# systemctl start frr
root@bordertor-11:mgmt:/var/home/cumulus#
root@bordertor-11:mgmt:/var/home/cumulus#
root@bordertor-11:mgmt:/var/home/cumulus# cat /var/lib/frr/.bgp_vni_rd.txt
1000114 3
1000112 4
1000113 5
104001 7
1000111 6
root@bordertor-11:mgmt:/var/home/cumulus# cat /var/lib/frr/.bgp_vrf_rd.txt
default 1
vrf2 9<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<vrf1 rd gone as expected
root@bordertor-11:mgmt:/var/home/cumulus# vtysh

bordertor-11# show bgp l2vpn evpn vni
Advertise Gateway Macip: Disabled
Advertise SVI Macip: Disabled
Advertise All VNI flag: Enabled
BUM flooding: Head-end replication
VXLAN flooding: Enabled
Number of L2 VNIs: 5
Number of L3 VNIs: 1
Flags: * - Kernel
VNI Type RD Import RT Export RT MAC-VRF Site-of-Origin Tenant VRF

1000114 L2 6.0.0.1:3 4640:1000114 4640:1000114 vrf1
1000112 L2 6.0.0.1:4 4640:1000112 4640:1000112 vrf1
1000113 L2 6.0.0.1:5 4640:1000113 4640:1000113 vrf2
104001 L2 6.0.0.1:7 4640:104001 4640:104001 vrf1
1000111 L2 6.0.0.1:6 4640:1000111 4640:1000111 vrf2
104002 L3 144.1.1.6:9 4640:104002 4640:104002 vrf2
--- RESULT 4a: PASS ---
104001 L2 with rd_id=7. VRF file has no vrf1 (BGP VRF instance removed).
vrf1 rd_id=2 bit is free. All L2 VNIs unchanged.
================================================================================

--- Step 4b: Add vrf1 back to config, restart FRR ---
bordertor-11# exit
root@bordertor-11:mgmt:/var/home/cumulus# vi /etc/frr/frr.conf<<<<<<<<<Add back vrf vrf1
root@bordertor-11:mgmt:/var/home/cumulus# systemctl restart frr
root@bordertor-11:mgmt:/var/home/cumulus# cat /var/lib/frr/.bgp_vrf_rd.txt
default 1
vrf2 9
vrf1 2<<<<<<<<<<<<Rd shows up again fro vrf 1
root@bordertor-11:mgmt:/var/home/cumulus# cat /var/lib/frr/.bgp_vni_rd.txt
1000114 3
1000112 4
1000113 5
1000111 6
root@bordertor-11:mgmt:/var/home/cumulus# vtysh

bordertor-11# show bgp l2vpn evpn vni
Advertise Gateway Macip: Disabled
Advertise SVI Macip: Disabled
Advertise All VNI flag: Enabled
BUM flooding: Head-end replication
VXLAN flooding: Enabled
Number of L2 VNIs: 4
Number of L3 VNIs: 2
Flags: * - Kernel
VNI Type RD Import RT Export RT MAC-VRF Site-of-Origin Tenant VRF

1000114 L2 6.0.0.1:3 4640:1000114 4640:1000114 vrf1
1000112 L2 6.0.0.1:4 4640:1000112 4640:1000112 vrf1
1000113 L2 6.0.0.1:5 4640:1000113 4640:1000113 vrf2
1000111 L2 6.0.0.1:6 4640:1000111 4640:1000111 vrf2
104001 L3 144.1.1.2:2 4640:104001 4640:104001 vrf1 <<<<<vrf1 with rd 2 reappears
104002 L3 144.1.1.6:9 4640:104002 4640:104002 vrf2
vrf1 got rd_id=2 (bf_assign_index gave lowest free bit).
104001 L3 with RD 144.1.1.2:2, orphan cleaned. All L2 VNIs unchanged.
================================================================================

bordertor-11# sh run
Building configuration...

Current configuration:
!
frr version 10.0.3
frr defaults datacenter
hostname bordertor-11
log file /var/log/frr/bgpd.log
log syslog informational
log timestamp precision 6
service integrated-vtysh-config
!
ip prefix-list LOCAL_HOST_VRF1 seq 1 permit 50.1.110.0/24
ip prefix-list LOCAL_HOST_VRF1 seq 2 deny any
ip prefix-list LOCAL_HOST_VRF2 seq 1 permit 60.1.110.0/24
ip prefix-list LOCAL_HOST_VRF2 seq 2 deny any
!
ipv6 prefix-list LOCAL_HOST_VRF1_v6 seq 1 permit 2050:1:1:110::/64
ipv6 prefix-list LOCAL_HOST_VRF1_v6 seq 2 deny any
ipv6 prefix-list LOCAL_HOST_VRF2_v6 seq 1 permit 2060:1:1:110::/64
ipv6 prefix-list LOCAL_HOST_VRF2_v6 seq 2 deny any
!
vrf mgmt
exit-vrf
!
vrf vrf1
vni 104001
exit-vrf
!
vrf vrf2
vni 104002
exit-vrf
!
router bgp 660000
bgp router-id 6.0.0.1
bgp graceful-restart
bgp bestpath as-path multipath-relax
bgp bestpath compare-routerid
neighbor TOR_LEAF_SPINE peer-group
neighbor TOR_LEAF_SPINE advertisement-interval 0
neighbor TOR_LEAF_SPINE timers 3 10
neighbor TOR_LEAF_SPINE timers connect 5
neighbor TOR_LEAF_SPINE capability extended-nexthop
neighbor swp1 interface peer-group TOR_LEAF_SPINE
neighbor swp1 remote-as external
neighbor swp1 advertisement-interval 0
neighbor swp1 timers 3 10
neighbor swp1 timers connect 5
neighbor swp2 interface peer-group TOR_LEAF_SPINE
neighbor swp2 remote-as external
neighbor swp2 advertisement-interval 0
neighbor swp2 timers 3 10
neighbor swp2 timers connect 5
!
address-family ipv4 unicast
redistribute connected route-map ALLOW_LOBR
neighbor TOR_LEAF_SPINE allowas-in
neighbor swp1 allowas-in
neighbor swp2 allowas-in
maximum-paths 16
maximum-paths ibgp 64
exit-address-family
!
address-family l2vpn evpn
neighbor TOR_LEAF_SPINE activate
advertise-all-vni
exit-address-family
exit
!
router bgp 660000 vrf vrf1
bgp router-id 144.1.1.2
no bgp network import-check
neighbor 144.1.1.1 remote-as external
neighbor 144.1.1.1 advertisement-interval 0
neighbor 144.1.1.1 timers 3 9
neighbor 144.1.1.1 timers connect 10
neighbor 155.1.1.1 remote-as external
neighbor 155.1.1.1 advertisement-interval 0
neighbor 155.1.1.1 timers 3 9
neighbor 155.1.1.1 timers connect 10
neighbor 2144:1:1:1::1 remote-as external
neighbor 2144:1:1:1::1 advertisement-interval 0
neighbor 2144:1:1:1::1 timers 3 9
neighbor 2144:1:1:1::1 timers connect 10
neighbor 2155:1:1:1::1 remote-as external
neighbor 2155:1:1:1::1 advertisement-interval 0
neighbor 2155:1:1:1::1 timers 3 9
neighbor 2155:1:1:1::1 timers connect 10
!
address-family ipv4 unicast
network 50.1.1.112/32
redistribute connected route-map HOST_ALLOW_1
maximum-paths 64
maximum-paths ibgp 64
exit-address-family
!
address-family ipv6 unicast
network 2050:1:1:1::112/128
redistribute connected route-map HOST_ALLOW_1_v6
neighbor 2144:1:1:1::1 activate
neighbor 2155:1:1:1::1 activate
maximum-paths 64
maximum-paths ibgp 64
exit-address-family
!
address-family l2vpn evpn
advertise ipv4 unicast
advertise ipv6 unicast
exit-address-family
exit
!
router bgp 660000 vrf vrf2
bgp router-id 144.1.1.6
no bgp network import-check
neighbor 144.1.1.5 remote-as external
neighbor 144.1.1.5 advertisement-interval 0
neighbor 144.1.1.5 timers 3 9
neighbor 144.1.1.5 timers connect 10
neighbor 155.1.1.5 remote-as external
neighbor 155.1.1.5 advertisement-interval 0
neighbor 155.1.1.5 timers 3 9
neighbor 155.1.1.5 timers connect 10
neighbor 2144:1:1:2::5 remote-as external
neighbor 2144:1:1:2::5 advertisement-interval 0
neighbor 2144:1:1:2::5 timers 3 9
neighbor 2144:1:1:2::5 timers connect 10
neighbor 2155:1:1:2::5 remote-as external
neighbor 2155:1:1:2::5 advertisement-interval 0
neighbor 2155:1:1:2::5 timers 3 9
neighbor 2155:1:1:2::5 timers connect 10
!
address-family ipv4 unicast
network 60.1.1.111/32
redistribute connected route-map HOST_ALLOW_2
maximum-paths 64
maximum-paths ibgp 64
exit-address-family
!
address-family ipv6 unicast
network 2060:1:1:1::111/128
redistribute connected route-map HOST_ALLOW_2_v6
neighbor 2144:1:1:2::5 activate
neighbor 2155:1:1:2::5 activate
maximum-paths 64
maximum-paths ibgp 64
exit-address-family
!
address-family l2vpn evpn
advertise ipv4 unicast
advertise ipv6 unicast
exit-address-family
exit
!
route-map ALLOW_LO permit 10
match interface lo
!
route-map ALLOW_LOBR permit 10
match interface lo
!
route-map ALLOW_LOBR permit 20
match interface br_l3vni
!
route-map HOST_ALLOW_1 permit 1
match ip address prefix-list LOCAL_HOST_VRF1
!
route-map HOST_ALLOW_1_v6 permit 1
match ipv6 address prefix-list LOCAL_HOST_VRF1_v6
!
route-map HOST_ALLOW_2 permit 1
match ip address prefix-list LOCAL_HOST_VRF2
!
route-map HOST_ALLOW_2_v6 permit 1
match ipv6 address prefix-list LOCAL_HOST_VRF2_v6
!
end

--- Second VNI check + state files ---
bordertor-11# show bgp l2vpn evpn vni
Advertise Gateway Macip: Disabled
Advertise SVI Macip: Disabled
Advertise All VNI flag: Enabled
BUM flooding: Head-end replication
VXLAN flooding: Enabled
Number of L2 VNIs: 4
Number of L3 VNIs: 2
Flags: * - Kernel
VNI Type RD Import RT Export RT MAC-VRF Site-of-Origin Tenant VRF

1000114 L2 6.0.0.1:3 4640:1000114 4640:1000114 vrf1
1000112 L2 6.0.0.1:4 4640:1000112 4640:1000112 vrf1
1000113 L2 6.0.0.1:5 4640:1000113 4640:1000113 vrf2
1000111 L2 6.0.0.1:6 4640:1000111 4640:1000111 vrf2
104001 L3 144.1.1.2:2 4640:104001 4640:104001 vrf1
104002 L3 144.1.1.6:9 4640:104002 4640:104002 vrf2
bordertor-11# exit
root@bordertor-11:mgmt:/var/home/cumulus# cat /var/lib/frr/.bgp_vni_rd.txt
1000114 3
1000112 4
1000113 5
1000111 6
root@bordertor-11:mgmt:/var/home/cumulus# cat /var/lib/frr/.bgp_vrf_rd.txt
default 1
vrf2 9
vrf1 2
================================================================================

================================================================================
TEST Runtime VTY removal — remove vrf1 via VTY commands at runtime
(no vni, no router bgp vrf vrf1), then restart FRR.
Expect: 104001 demotes L3->L2 with rd_id=7, vrf1 removed from VRF
file. After restart, config still has vrf1 (frr.conf not
changed via "write memory" yet), so vrf1 comes back with
rd_id=2.

root@bordertor-11:mgmt:/var/home/cumulus# exit
exit
cumulus@bordertor-11:mgmt:~$ sudo bash
root@bordertor-11:mgmt:/var/home/cumulus# vtysh

--- Step 6a: Runtime VTY removal of vrf1 ---
bordertor-11# conf
bordertor-11(config)# no router bgp 660000 vrf vrf1
% Please unconfigure l3vni 104001
bordertor-11(config)# vrf vrf1
bordertor-11(config-vrf)# no vni 104001
bordertor-11(config-vrf)# exit
bordertor-11(config)# no router bgp 660000 vrf vrf1
bordertor-11(config)# exit
bordertor-11# show bgp l2vpn evpn vni
Advertise Gateway Macip: Disabled
Advertise SVI Macip: Disabled
Advertise All VNI flag: Enabled
BUM flooding: Head-end replication
VXLAN flooding: Enabled
Number of L2 VNIs: 5
Number of L3 VNIs: 1
Flags: * - Kernel
VNI Type RD Import RT Export RT MAC-VRF Site-of-Origin Tenant VRF

1000114 L2 6.0.0.1:3 4640:1000114 4640:1000114 vrf1
1000112 L2 6.0.0.1:4 4640:1000112 4640:1000112 vrf1
1000113 L2 6.0.0.1:5 4640:1000113 4640:1000113 vrf2
104001 L2 6.0.0.1:7 4640:104001 4640:104001 vrf1
1000111 L2 6.0.0.1:6 4640:1000111 4640:1000111 vrf2 <<<<<<<vrf 1 disappeared
104002 L3 144.1.1.6:9 4640:104002 4640:104002 vrf2
bordertor-11# exit
root@bordertor-11:mgmt:/var/home/cumulus# cat /var/lib/frr/.bgp_vrf_rd.txt
default 1
vrf2 9<<<<<<<vrf 1 disappeared
root@bordertor-11:mgmt:/var/home/cumulus# cat /var/lib/frr/.bgp_vni_rd.txt
1000114 3
1000112 4
1000113 5
1000111 6
104001 7

Runtime VTY deletion works: 104001 L2 with rd_id=7, vrf1 removed from
VRF file, 104001=7 added to VNI file. All L2 VNIs unchanged.
================================================================================

--- Step 6b: Restart FRR (frr.conf still has vrf1 — not saved in config file) ---
root@bordertor-11:mgmt:/var/home/cumulus# systemctl resatrt frr
Unknown command verb resatrt.
root@bordertor-11:mgmt:/var/home/cumulus# systemctl restart frr
root@bordertor-11:mgmt:/var/home/cumulus# cat /var/lib/frr/.bgp_vrf_rd.txt
default 1
vrf2 9
vrf1 2<<<<<<<<as we didn’t do write memory, it rightly reappeared
root@bordertor-11:mgmt:/var/home/cumulus# cat /var/lib/frr/.bgp_vni_rd.txt
1000114 3
1000112 4
1000113 5
1000111 6

--- Verify: "sh run" shows vrf1 restored from frr.conf (was not saved) ---
root@bordertor-11:mgmt:/var/home/cumulus# sh run
sh: 0: cannot open run: No such file
root@bordertor-11:mgmt:/var/home/cumulus# vtysh

bordertor-11# sh run
Building configuration...

Current configuration:
!
frr version 10.0.3
frr defaults datacenter
hostname bordertor-11
log file /var/log/frr/bgpd.log
log syslog informational
log timestamp precision 6
service integrated-vtysh-config
!
ip prefix-list LOCAL_HOST_VRF1 seq 1 permit 50.1.110.0/24
ip prefix-list LOCAL_HOST_VRF1 seq 2 deny any
ip prefix-list LOCAL_HOST_VRF2 seq 1 permit 60.1.110.0/24
ip prefix-list LOCAL_HOST_VRF2 seq 2 deny any
!
ipv6 prefix-list LOCAL_HOST_VRF1_v6 seq 1 permit 2050:1:1:110::/64
ipv6 prefix-list LOCAL_HOST_VRF1_v6 seq 2 deny any
ipv6 prefix-list LOCAL_HOST_VRF2_v6 seq 1 permit 2060:1:1:110::/64
ipv6 prefix-list LOCAL_HOST_VRF2_v6 seq 2 deny any
!
vrf mgmt
exit-vrf
!
vrf vrf1
vni 104001
exit-vrf
!
vrf vrf2
vni 104002
exit-vrf
!
router bgp 660000
bgp router-id 6.0.0.1
bgp graceful-restart
bgp bestpath as-path multipath-relax
bgp bestpath compare-routerid
neighbor TOR_LEAF_SPINE peer-group
neighbor TOR_LEAF_SPINE advertisement-interval 0
neighbor TOR_LEAF_SPINE timers 3 10
neighbor TOR_LEAF_SPINE timers connect 5
neighbor TOR_LEAF_SPINE capability extended-nexthop
neighbor swp1 interface peer-group TOR_LEAF_SPINE
neighbor swp1 remote-as external
neighbor swp1 advertisement-interval 0
neighbor swp1 timers 3 10
neighbor swp1 timers connect 5
neighbor swp2 interface peer-group TOR_LEAF_SPINE
neighbor swp2 remote-as external
neighbor swp2 advertisement-interval 0
neighbor swp2 timers 3 10
neighbor swp2 timers connect 5
!
address-family ipv4 unicast
redistribute connected route-map ALLOW_LOBR
neighbor TOR_LEAF_SPINE allowas-in
neighbor swp1 allowas-in
neighbor swp2 allowas-in
maximum-paths 16
maximum-paths ibgp 64
exit-address-family
!
address-family l2vpn evpn
neighbor TOR_LEAF_SPINE activate
advertise-all-vni
exit-address-family
exit
!
router bgp 660000 vrf vrf1
bgp router-id 144.1.1.2
no bgp network import-check
neighbor 144.1.1.1 remote-as external
neighbor 144.1.1.1 advertisement-interval 0
neighbor 144.1.1.1 timers 3 9
neighbor 144.1.1.1 timers connect 10
neighbor 155.1.1.1 remote-as external
neighbor 155.1.1.1 advertisement-interval 0
neighbor 155.1.1.1 timers 3 9
neighbor 155.1.1.1 timers connect 10
neighbor 2144:1:1:1::1 remote-as external
neighbor 2144:1:1:1::1 advertisement-interval 0
neighbor 2144:1:1:1::1 timers 3 9
neighbor 2144:1:1:1::1 timers connect 10
neighbor 2155:1:1:1::1 remote-as external
neighbor 2155:1:1:1::1 advertisement-interval 0
neighbor 2155:1:1:1::1 timers 3 9
neighbor 2155:1:1:1::1 timers connect 10
!
address-family ipv4 unicast
network 50.1.1.112/32
redistribute connected route-map HOST_ALLOW_1
maximum-paths 64
maximum-paths ibgp 64
exit-address-family
!
address-family ipv6 unicast
network 2050:1:1:1::112/128
redistribute connected route-map HOST_ALLOW_1_v6
neighbor 2144:1:1:1::1 activate
neighbor 2155:1:1:1::1 activate
maximum-paths 64
maximum-paths ibgp 64
exit-address-family
!
address-family l2vpn evpn
advertise ipv4 unicast
advertise ipv6 unicast
exit-address-family
exit
!
router bgp 660000 vrf vrf2
bgp router-id 144.1.1.6
no bgp network import-check
neighbor 144.1.1.5 remote-as external
neighbor 144.1.1.5 advertisement-interval 0
neighbor 144.1.1.5 timers 3 9
neighbor 144.1.1.5 timers connect 10
neighbor 155.1.1.5 remote-as external
neighbor 155.1.1.5 advertisement-interval 0
neighbor 155.1.1.5 timers 3 9
neighbor 155.1.1.5 timers connect 10
neighbor 2144:1:1:2::5 remote-as external
neighbor 2144:1:1:2::5 advertisement-interval 0
neighbor 2144:1:1:2::5 timers 3 9
neighbor 2144:1:1:2::5 timers connect 10
neighbor 2155:1:1:2::5 remote-as external
neighbor 2155:1:1:2::5 advertisement-interval 0
neighbor 2155:1:1:2::5 timers 3 9
neighbor 2155:1:1:2::5 timers connect 10
!
address-family ipv4 unicast
network 60.1.1.111/32
redistribute connected route-map HOST_ALLOW_2
maximum-paths 64
maximum-paths ibgp 64
exit-address-family
!
address-family ipv6 unicast
network 2060:1:1:1::111/128
redistribute connected route-map HOST_ALLOW_2_v6
neighbor 2144:1:1:2::5 activate
neighbor 2155:1:1:2::5 activate
maximum-paths 64
maximum-paths ibgp 64
exit-address-family
!
address-family l2vpn evpn
advertise ipv4 unicast
advertise ipv6 unicast
exit-address-family
exit
!
route-map ALLOW_LO permit 10
match interface lo
!
route-map ALLOW_LOBR permit 10
match interface lo
!
route-map ALLOW_LOBR permit 20
match interface br_l3vni
!
route-map HOST_ALLOW_1 permit 1
match ip address prefix-list LOCAL_HOST_VRF1
!
route-map HOST_ALLOW_1_v6 permit 1
match ipv6 address prefix-list LOCAL_HOST_VRF1_v6
!
route-map HOST_ALLOW_2 permit 1
match ip address prefix-list LOCAL_HOST_VRF2
!
route-map HOST_ALLOW_2_v6 permit 1
match ipv6 address prefix-list LOCAL_HOST_VRF2_v6
!
end
bordertor-11# conf
bordertor-11(config)# exit
bordertor-11# show bgp l2vpn evpn vni
Advertise Gateway Macip: Disabled
Advertise SVI Macip: Disabled
Advertise All VNI flag: Enabled
BUM flooding: Head-end replication
VXLAN flooding: Enabled
Number of L2 VNIs: 4
Number of L3 VNIs: 2
Flags: * - Kernel
VNI Type RD Import RT Export RT MAC-VRF Site-of-Origin Tenant VRF

1000114 L2 6.0.0.1:3 4640:1000114 4640:1000114 vrf1
1000112 L2 6.0.0.1:4 4640:1000112 4640:1000112 vrf1
1000113 L2 6.0.0.1:5 4640:1000113 4640:1000113 vrf2
1000111 L2 6.0.0.1:6 4640:1000111 4640:1000111 vrf2
104001 L3 144.1.1.2:2 4640:104001 4640:104001 vrf1
104002 L3 144.1.1.6:9 4640:104002 4640:104002 vrf2
bordertor-11# exit
root@bordertor-11:mgmt:/var/home/cumulus# cat /var/lib/frr/.bgp_vni_rd.txt
1000114 3
1000112 4
1000113 5
1000111 6
root@bordertor-11:mgmt:/var/home/cumulus# cat /var/lib/frr/.bgp_vrf_rd.txt
default 1
vrf2 9
vrf1 2

vrf1 restored from frr.conf with rd_id=2. 104001 back to L3 with
RD 144.1.1.2:2. Orphan cleaned. All L2 VNIs unchanged.
================================================================================

================================================================================
TEST : VTY removal + "write memory" + restart + VTY re-add.
This tests the full operational workflow:
1) Remove vrf1 via VTY at runtime
2) "write memory" to persist removal to frr.conf
3) Restart FRR (config has no vrf1)
4) Re-add vrf1 via VTY at runtime
Expect: Step 3 keeps 104001 as L2 (rd_id=7) since config lacks vrf1.
Step 4 promotes 104001 back to L3 with rd_id=2 (new assignment).

--- Step 7a: Runtime VTY removal + write memory ---
root@bordertor-11:mgmt:/var/home/cumulus# exit
exit
cumulus@bordertor-11:mgmt:~$ sudo bash
root@bordertor-11:mgmt:/var/home/cumulus# vtysh

bordertor-11# conf
bordertor-11(config)# vrf vrf1
bordertor-11(config-vrf)# no vni 104001
bordertor-11(config-vrf)# exit
bordertor-11(config)# no router bgp 660000 vrf vrf1
bordertor-11(config)# exit
bordertor-11# write memory
Note: this version of vtysh never writes vtysh.conf
Building Configuration...
Integrated configuration saved to /etc/frr/frr.conf
[OK]
bordertor-11# exit
root@bordertor-11:mgmt:/var/home/cumulus# vtysh

bordertor-11# show bgp l2vpn evpn vni
Advertise Gateway Macip: Disabled
Advertise SVI Macip: Disabled
Advertise All VNI flag: Enabled
BUM flooding: Head-end replication
VXLAN flooding: Enabled
Number of L2 VNIs: 5
Number of L3 VNIs: 1
Flags: * - Kernel
VNI Type RD Import RT Export RT MAC-VRF Site-of-Origin Tenant VRF

1000114 L2 6.0.0.1:3 4640:1000114 4640:1000114 vrf1
1000112 L2 6.0.0.1:4 4640:1000112 4640:1000112 vrf1
1000113 L2 6.0.0.1:5 4640:1000113 4640:1000113 vrf2
104001 L2 6.0.0.1:7 4640:104001 4640:104001 vrf1
1000111 L2 6.0.0.1:6 4640:1000111 4640:1000111 vrf2
104002 L3 144.1.1.6:9 4640:104002 4640:104002 vrf2
bordertor-11# exit
root@bordertor-11:mgmt:/var/home/cumulus# cat /var/lib/frr/.bgp_vrf_rd.txt
default 1
vrf2 9<<<<<vrf 1 disappeared
root@bordertor-11:mgmt:/var/home/cumulus# cat /var/lib/frr/.bgp_vni_rd.txt
1000114 3
1000112 4
1000113 5
1000111 6
104001 7

VTY removal + write memory: 104001 L2 with rd_id=7, vrf1 removed from
VRF file, config saved to frr.conf without vrf1.
================================================================================

Restart FRR (frr.conf has no vrf1, saved by write memory) ---
root@bordertor-11:mgmt:/var/home/cumulus# systemctl restart frr
root@bordertor-11:mgmt:/var/home/cumulus#
root@bordertor-11:mgmt:/var/home/cumulus# cat /var/lib/frr/.bgp_vrf_rd.txt
default 1
vrf2 9<<<<<vrf 1 doesn’t show up anymore
root@bordertor-11:mgmt:/var/home/cumulus# cat /var/lib/frr/.bgp_vni_rd.txt
1000114 3
1000112 4
1000113 5
1000111 6
104001 7
root@bordertor-11:mgmt:/var/home/cumulus# vtysh

bordertor-11# show bgp l2vpn evpn vni
Advertise Gateway Macip: Disabled
Advertise SVI Macip: Disabled
Advertise All VNI flag: Enabled
BUM flooding: Head-end replication
VXLAN flooding: Enabled
Number of L2 VNIs: 5
Number of L3 VNIs: 1
Flags: * - Kernel
VNI Type RD Import RT Export RT MAC-VRF Site-of-Origin Tenant VRF

1000114 L2 6.0.0.1:3 4640:1000114 4640:1000114 vrf1
1000112 L2 6.0.0.1:4 4640:1000112 4640:1000112 vrf1
1000113 L2 6.0.0.1:5 4640:1000113 4640:1000113 vrf2
104001 L2 6.0.0.1:7 4640:104001 4640:104001 vrf1
1000111 L2 6.0.0.1:6 4640:1000111 4640:1000111 vrf2
104002 L3 144.1.1.6:9 4640:104002 4640:104002 vrf2

Restart with saved config: 104001 stays L2 with rd_id=7 (persistent).
No vrf1 in VRF file (correctly absent). VNI 104001=7 persisted.
================================================================================

--- Step 7c: Re-add vrf1 + full BGP config via VTY at runtime ---
bordertor-11# conf
bordertor-11(config)# vrf vrf1
bordertor-11(config-vrf)# vni 104001
bordertor-11(config-vrf)# exit-vrf
bordertor-11(config)# !
bordertor-11(config)# router bgp 660000 vrf vrf1
bordertor-11(config-router)# bgp router-id 144.1.1.2
bordertor-11(config-router)# no bgp network import-check
bordertor-11(config-router)# neighbor 144.1.1.1 remote-as external
bordertor-11(config-router)# neighbor 144.1.1.1 advertisement-interval 0
bordertor-11(config-router)# neighbor 144.1.1.1 timers 3 9
bordertor-11(config-router)# neighbor 144.1.1.1 timers connect 10
bordertor-11(config-router)# neighbor 155.1.1.1 remote-as external
bordertor-11(config-router)# neighbor 155.1.1.1 advertisement-interval 0
bordertor-11(config-router)# neighbor 155.1.1.1 timers 3 9
bordertor-11(config-router)# neighbor 155.1.1.1 timers connect 10
bordertor-11(config-router)# neighbor 2144:1:1:1::1 remote-as external
bordertor-11(config-router)# neighbor 2144:1:1:1::1 advertisement-interval 0
bordertor-11(config-router)# neighbor 2144:1:1:1::1 timers 3 9
bordertor-11(config-router)# neighbor 2144:1:1:1::1 timers connect 10
bordertor-11(config-router)# neighbor 2155:1:1:1::1 remote-as external
bordertor-11(config-router)# neighbor 2155:1:1:1::1 advertisement-interval 0
bordertor-11(config-router)# neighbor 2155:1:1:1::1 timers 3 9
bordertor-11(config-router)# neighbor 2155:1:1:1::1 timers connect 10
bordertor-11(config-router)# !
bordertor-11(config-router)# address-family ipv4 unicast
bordertor-11(config-router-af)# network 50.1.1.112/32
bordertor-11(config-router-af)# redistribute connected route-map HOST_ALLOW_1
bordertor-11(config-router-af)# maximum-paths 64
bordertor-11(config-router-af)# maximum-paths ibgp 64
bordertor-11(config-router-af)# exit-address-family
bordertor-11(config-router)# !
bordertor-11(config-router)# address-family ipv6 unicast
bordertor-11(config-router-af)# network 2050:1:1:1::112/128
bordertor-11(config-router-af)# redistribute connected route-map HOST_ALLOW_1_v6
bordertor-11(config-router-af)# neighbor 2144:1:1:1::1 activate
bordertor-11(config-router-af)# neighbor 2155:1:1:1::1 activate
bordertor-11(config-router-af)# maximum-paths 64
bordertor-11(config-router-af)# maximum-paths ibgp 64
bordertor-11(config-router-af)# exit-address-family
bordertor-11(config-router)# !
bordertor-11(config-router)# address-family l2vpn evpn
bordertor-11(config-router-af)# advertise ipv4 unicast
bordertor-11(config-router-af)# advertise ipv6 unicast
bordertor-11(config-router-af)# exit-address-family
bordertor-11(config-router)# exit
bordertor-11(config)# !
bordertor-11(config)# exit
bordertor-11# show bgp l2vpn evpn vni
Advertise Gateway Macip: Disabled
Advertise SVI Macip: Disabled
Advertise All VNI flag: Enabled
BUM flooding: Head-end replication
VXLAN flooding: Enabled
Number of L2 VNIs: 4
Number of L3 VNIs: 2
Flags: * - Kernel
VNI Type RD Import RT Export RT MAC-VRF Site-of-Origin Tenant VRF

1000114 L2 6.0.0.1:3 4640:1000114 4640:1000114 vrf1
1000112 L2 6.0.0.1:4 4640:1000112 4640:1000112 vrf1
1000113 L2 6.0.0.1:5 4640:1000113 4640:1000113 vrf2
1000111 L2 6.0.0.1:6 4640:1000111 4640:1000111 vrf2
104002 L3 144.1.1.6:9 4640:104002 4640:104002 vrf2
104001 L3 144.1.1.2:2 4640:104001 4640:104001 vrf1 <<<vrf1 came back
bordertor-11# exit
root@bordertor-11:mgmt:/var/home/cumulus# cat /var/lib/frr/.bgp_vni_rd.txt
1000114 3
1000112 4
1000113 5
1000111 6
root@bordertor-11:mgmt:/var/home/cumulus# cat /var/lib/frr/.bgp_vrf_rd.txt
default 1
vrf2 9
vrf1 2

VTY re-add: vrf1 got rd_id=2 (bf_assign_index gave lowest free bit).
104001 promoted L2->L3 with RD 144.1.1.2:2. Orphan "104001 7" cleaned.
All L2 VNIs unchanged.
================================================================================

================================================================================
TEST : Final stability — "write memory" to persist re-added vrf1,
restart FRR, verify RDs are identical.
Expect: All RDs unchanged after restart. Final steady state.

root@bordertor-11:mgmt:/var/home/cumulus# vtysh

bordertor-11# write memory
Note: this version of vtysh never writes vtysh.conf
Building Configuration...
Integrated configuration saved to /etc/frr/frr.conf
[OK]
bordertor-11# vtysh
% Unknown command: vtysh
bordertor-11# exit
root@bordertor-11:mgmt:/var/home/cumulus# systemctl restart frr
root@bordertor-11:mgmt:/var/home/cumulus# cat /var/lib/frr/.bgp_vrf_rd.txt
default 1
vrf2 9
vrf1 2
root@bordertor-11:mgmt:/var/home/cumulus# cat /var/lib/frr/.bgp_vni_rd.txt
1000114 3
1000112 4
1000113 5
1000111 6
root@bordertor-11:mgmt:/var/home/cumulus# vtysh

--- First check after restart ---
bordertor-11# show bgp l2vpn evpn vni
Advertise Gateway Macip: Disabled
Advertise SVI Macip: Disabled
Advertise All VNI flag: Enabled
BUM flooding: Head-end replication
VXLAN flooding: Enabled
Number of L2 VNIs: 4
Number of L3 VNIs: 2
Flags: * - Kernel
VNI Type RD Import RT Export RT MAC-VRF Site-of-Origin Tenant VRF

1000114 L2 6.0.0.1:3 4640:1000114 4640:1000114 vrf1
1000112 L2 6.0.0.1:4 4640:1000112 4640:1000112 vrf1
1000113 L2 6.0.0.1:5 4640:1000113 4640:1000113 vrf2
1000111 L2 6.0.0.1:6 4640:1000111 4640:1000111 vrf2
104002 L3 144.1.1.6:9 4640:104002 4640:104002 vrf2
104001 L3 144.1.1.2:2 4640:104001 4640:104001 vrf1

--- Second check (idempotency verification) ---
bordertor-11# show bgp l2vpn evpn vni
Advertise Gateway Macip: Disabled
Advertise SVI Macip: Disabled
Advertise All VNI flag: Enabled
BUM flooding: Head-end replication
VXLAN flooding: Enabled
Number of L2 VNIs: 4
Number of L3 VNIs: 2
Flags: * - Kernel
VNI Type RD Import RT Export RT MAC-VRF Site-of-Origin Tenant VRF

1000114 L2 6.0.0.1:3 4640:1000114 4640:1000114 vrf1
1000112 L2 6.0.0.1:4 4640:1000112 4640:1000112 vrf1
1000113 L2 6.0.0.1:5 4640:1000113 4640:1000113 vrf2
1000111 L2 6.0.0.1:6 4640:1000111 4640:1000111 vrf2
104002 L3 144.1.1.6:9 4640:104002 4640:104002 vrf2
104001 L3 144.1.1.2:2 4640:104001 4640:104001 vrf1
bordertor-11#

Both checks identical. Final steady state achieved.
All RDs stable: vrf1=2, vrf2=9, 1000114=3, 1000112=4, 1000113=5, 1000111=6
104001 L3 inheriting vrf1 rd_id=2, 104002 L3 inheriting vrf2 rd_id=9

souroy@souroy-mlt ~ %

greptile-apps · 2026-03-19T23:41:50Z

Greptile Summary

This PR persists EVPN Route Distinguisher index assignments (VRF-name → rd_id and VNI → rd_id) to lightweight state files under frr_libstatedir, so that bgpd restores the same rd_id values after a restart rather than re-allocating them non-deterministically. The design is sound: state files are loaded before config processing, the rd_idspace bitfield is pre-seeded from the files to prevent collisions, hash lookups on VRF/VNI creation choose restore-over-allocate, and a bgp_config_end hook prunes orphan entries. Atomic writes via tmp-file + rename() guard against corruption.

Key changes:

bgpd/bgpd.h: Two new typesafe hashes (vrf_rd_state_hash, vni_rd_state_hash) and entry structs added to bgp_master.
bgpd/bgpd.c: Load, append, rewrite, and orphan-cleanup functions for the VRF state file; integration at VRF create/delete and bgp_master_init.
bgpd/bgp_evpn.c: Analogous load/assign/rewrite for the VNI state file; integrated into bgp_evpn_new and bgp_evpn_free.
bgpd/bgp_main.c: Correctly frees both hash tables at process exit using typesafe hash _pop (addressing the null-free-callback issue from the prior review round).
tests/topotests/bgp_evpn_gr/: Three new topotests covering state-file creation, RD persistence across daemon restart, and orphan cleanup.

Remaining concerns from prior rounds:

bgp_evpn_log_vni_rd_to_statefile still omits the mkdir(frr_libstatedir) call that the VRF append path includes — VNI RD persistence silently fails on a fresh install before the state directory exists.
O(n) full file rewrites on each individual VNI deletion in bgp_evpn_free (acknowledged with a TODO comment).

New finding: Both bgp_rewrite_vrf_rd_statefile and bgp_evpn_rewrite_vni_rd_statefile silently discard fclose failures without logging, unlike the rename failure path directly below them.

Confidence Score: 3/5

PR is functionally correct and the core restart-stability goal is proven by UT logs, but two issues from prior review rounds remain unresolved.
Two prior-round concerns are now fixed (null free-callback replaced with typesafe hash pop; dead hash_vrf_rd_state_entry_free removed). However, bgp_evpn_log_vni_rd_to_statefile still omits mkdir (meaning VNI RD persistence silently fails on any system where /var/lib/frr does not yet exist), and O(n) full-rewrite-per-VNI-deletion is still present. The new silent fclose failure is P2. Score reflects convergence progress but two residual items from the prior review still warrant a targeted fix pass.
bgpd/bgp_evpn.c (bgp_evpn_log_vni_rd_to_statefile missing mkdir) and bgpd/bgpd.c + bgpd/bgp_evpn.c rewrite functions (silent fclose failure)

Important Files Changed

Filename	Overview
bgpd/bgpd.c	Core of the RD caching feature: adds load/save/rewrite functions for the VRF state file, `bgp_assign_or_restore_vrf_rd_id`, orphan cleanup hook, and integration into `bgp_free`. Logic is sound; initialization order is correct (rd_idspace initialized before statefile load). Silent fclose failure in `bgp_rewrite_vrf_rd_statefile` is the main new concern. The `cleanup_done` static bool is intentional: orphan cleanup is a one-time startup pass and all runtime deletions are handled by the `bgp_free` path.
bgpd/bgp_evpn.c	Adds VNI-side RD caching: `bgp_evpn_assign_rd_id_for_vni`, `bgp_evpn_log_vni_rd_to_statefile`, and `bgp_evpn_rewrite_vni_rd_statefile`. The append function still lacks `mkdir` (flagged in prior review thread). The rewrite function's `fclose` failure is also silent. O(n) per-VNI full file rewrite in `bgp_evpn_free` is acknowledged with a TODO comment.
bgpd/bgpd.h	Adds `PREDECL_HASH`/`DECLARE_HASH` declarations for `vrf_rd_state_hash` and `vni_rd_state_hash`, the two entry structs, and their hash/compare inline functions. Hash functions and comparators are correct: `vni_rd_state_hash_cmp` uses unsigned subtraction cast to int which is safe as an equality test for the typesafe hash. State file name macros are clean.
bgpd/bgp_main.c	Adds cleanup of both hash tables in `bgp_exit` using typesafe hash `_pop` iteration, correctly freeing the `name` field for VRF entries and the entry structs. Memory management is correct; this addresses the null free-callback concern from the prior review round.
tests/topotests/bgp_evpn_gr/test_bgp_evpn_gr.py	Adds three new topotests: state file creation, RD persistence across restart, and orphan cleanup. Test logic is generally sound — startup correctly uses `start_router_daemons` + explicit `vtysh -f` so each scenario loads exactly the intended config. Minor: `PE1_minimal.conf` is written to the test source directory (`CWD`) and never cleaned up, which can leave stale files in the repo tree across CI runs.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[bgp_master_init] --> B[bf_init rd_idspace]
    B --> C[bgp_load_vrf_rd_statefile\nparse file → hash, bf_set_bit]
    C --> D[bgp_load_vni_rd_statefile\nparse file → hash, bf_set_bit]
    D --> E[config loaded via frr.conf / vtysh]

    E --> F{VRF created\nbgp_create}
    F -->|in hash?| G[restore rd_id\nbf_set_bit if needed\nmark used=true]
    F -->|not found| H[bf_assign_index\nappend to .bgp_vrf_rd.txt\nadd to hash used=true]

    E --> I{VNI created\nbgp_evpn_new}
    I -->|in hash?| J[restore rd_id\nbf_set_bit if needed\nmark used=true]
    I -->|not found| K[bf_assign_index\nappend to .bgp_vni_rd.txt\nadd to hash used=true]

    E --> L[bgp_config_end hook\nbgp_rd_state_cleanup_after_config]
    L --> M{iterate hashes\nused == false?}
    M -->|yes = orphan| N[bf_release_index\nremove from hash\nrewrite state file]
    M -->|no| O[keep entry]

    P[Runtime VRF delete\nbgp_free] --> Q[remove from hash\nif !terminating: rewrite VRF file\nbf_release_index]
    R[Runtime VNI delete\nbgp_evpn_free] --> S[remove from hash\nif !terminating: rewrite VNI file\nbf_release_index]

    T[SIGTERM shutdown\nbm→terminating=true] --> U[bgp_free per instance\nremove from hash\nNO file rewrite - preserve on disk]
    U --> V[bgp_exit\nfree remaining hash memory]

Prompt To Fix All With AI

This is a comment left during a code review.
Path: bgpd/bgpd.c
Line: 242-254

Comment:
**Silent fclose failure swallows I/O errors**

`bgp_rewrite_vrf_rd_statefile` removes the tmp file on `fclose` failure but emits no log message, making disk-full or I/O errors invisible. The `bgp_evpn_rewrite_vni_rd_statefile` function in `bgp_evpn.c` has the same gap. Consider logging an error here, matching the `zlog_err` you already emit for a failed `rename`:

```suggestion
	if (fclose(fp) != 0) {
		zlog_err("BGP: failed to flush %s: %s", tmp, safe_strerror(errno));
		remove(tmp);
		return;
	}
```

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: bgpd/bgpd.c
Line: 256-259

Comment:
**Silent fclose failure in VRF state rewrite**

When `fclose` fails (e.g., disk full, I/O error), the tmp file is silently removed and the function returns. There is no `zlog_err` here, unlike the `rename` path directly below. This makes diagnosing persistence failures difficult. The same gap exists in `bgp_evpn_rewrite_vni_rd_statefile` at `bgp_evpn.c:182`.

```suggestion
	if (fclose(fp) != 0) {
		zlog_err("BGP: failed to flush %s: %s", tmp, safe_strerror(errno));
		remove(tmp);
		return;
	}
```

How can I resolve this? If you propose a fix, please make it concise.

_{Reviews (2): Last reviewed commit: "tests: add topotests for RD caching pers..." | Re-trigger Greptile}

bgpd/bgp_main.c

bgpd/bgp_evpn.c

bgpd/bgpd.c

ton31337 · 2026-03-20T07:27:49Z

What if we have a knob to specify the type for RD encoding (e.g.: evpn rd-format type (0-2))? Yes, that has a limitation too for ASN, but maybe that would be enough in most of the cases?

Or if this is a "requirement" (type 1 with the loopback), then isn't it possible to carry VNI as a large/extended community instead of carrying in NLRI?

mjstapp

this is exactly the sort of change that really must have a design for review - adding statefulness to the bgp daemon is a significant change. it seems (at first look) out of proportion to the issue described. if unpredictable values are a problem ... configure stable values as part of provisioning/orchestration. if the existing format is an issue, then ... consider other formats that might work better. those are the kinds of things that should be explored in a design proposal.

soumyar-roy · 2026-03-23T21:21:46Z

ci:rerun

soumyar-roy · 2026-03-24T05:55:43Z

this is exactly the sort of change that really must have a design for review - adding statefulness to the bgp daemon is a significant change. it seems (at first look) out of proportion to the issue described. if unpredictable values are a problem ... configure stable values as part of provisioning/orchestration. if the existing format is an issue, then ... consider other formats that might work better. those are the kinds of things that should be explored in a design proposal.

I have added a design summary "# Design Proposal: Persistent Route Distinguisher Index for EVPN Graceful Restart", just after initial description.

soumyar-roy · 2026-03-24T06:27:24Z

What if we have a knob to specify the type for RD encoding (e.g.: evpn rd-format type (0-2))? Yes, that has a limitation too for ASN, but maybe that would be enough in most of the cases?

Type 0 (ASN:32bit) could embed the VNI directly in the 32-bit value field, making L2VNI RDs deterministic. But RDs must be unique per PE, not just per VNI. Two PEs in the same AS advertising VNI 100 would both produce 65000:100 — identical RDs.To work around that with Type 0, you'd need to encode both a per-router differentiator and the VNI in 32 bits. VNIs are 24-bit, leaving only 8 bits for router discrimination — enough for 256 PEs, which may not scale.

Also for l3vrf, VNI may not be available, so sometime no key/identifier to derive RD

Or if this is a "requirement" (type 1 with the loopback), then isn't it possible to carry VNI as a large/extended community instead of carrying in NLRI?
Not sure I understood this, but just to note VNI are not embedded in RD, but yes can be used to derive RD index(Obviously VNIs are 24-bit, they can't fit in the 16-bit field). So VNI doesn't have any binding on RD, its more implementation choice to derive RD from it. VNIs are are anyway already sent via NLRI labels/attribute etc

I have added the detail in "# Design Proposal: Persistent Route Distinguisher Index for EVPN Graceful Restart"

ton31337 · 2026-03-24T09:50:07Z

Let's say we have a new extended community (EC) that carries VNI. Isn't possible to do something like this?

Reserve 0 in Type 1 RDs as a sentinel meaning "VNI carried in VNI Extended Community" (0 is a placeholder, e.g.: 10.0.0.1:0);
Define the VNI Extended Community as above;
Define the auto-derivation rule: RD = :0, VNI EC attached to every EVPN route.

soumyar-roy · 2026-03-24T17:07:31Z

@greptile check if there is any other memory leak

greptile-apps · 2026-03-24T17:08:59Z

No, I don't see any other memory leaks in the code.

The memory management looks correct:

VRF entries:

Allocated in bgp_load_vrf_rd_statefile() and bgp_assign_or_restore_vrf_rd_id()
Freed in bgp_free(), bgp_rd_state_cleanup_after_config(), and bgp_exit()
Duplicates during load are immediately freed (both name and entry)

VNI entries:

Allocated in bgp_load_vni_rd_statefile() and bgp_evpn_assign_rd_id_for_vni()
Freed in bgp_evpn_free(), bgp_rd_state_cleanup_after_config(), and bgp_exit()
Duplicates during load are immediately freed

File handles:

All fopen calls have corresponding fclose calls
Error paths only return after failed fopen (no leak) or after successful fclose

All entries added to the hash tables are eventually removed and freed, either during normal operation (VRF/VNI deletion), orphan cleanup, or at shutdown via hash_pop loops in bgp_exit().

sudharr86 · 2026-03-24T18:03:44Z

Let's say we have a new extended community (EC) that carries VNI. Isn't possible to do something like this?

Reserve 0 in Type 1 RDs as a sentinel meaning "VNI carried in VNI Extended Community" (0 is a placeholder, e.g.: 10.0.0.1:0);

Define the VNI Extended Community as above;

Define the auto-derivation rule: RD = :0, VNI EC attached to every EVPN route.

VNI is already part of the NLRI. and is being used for downstream VNI scenarios. its already carried in MP_REACH_NLRI. it doesnt have to be part of RD
Also a new EC will be redundant

soumyar-roy · 2026-03-24T18:40:54Z

Confirmed OSPF writes to same location for states i,e at frr_libstatedir, where this PR also stores state for RDs.
#define OSPFD_STATE_NAME "%s/ospfd.json", frr_libstatedir
#define OSPFD_INST_STATE_NAME(i) "%s/ospfd-%d.json", frr_libstatedir,

soumyar-roy · 2026-03-24T21:56:16Z

ci:rerun

1. Fix memory leak on duplicate hash entries: when loading VRF/VNI state files, duplicate entries could cause the newly allocated entry to be silently ignored. Free the unused allocation when a duplicate is detected. 2. Reserve rd_id bits during statefile load: call bf_set_bit() while reading state files so cached rd_ids are not reassigned to new VRFs or VNIs. 3. Replace line-by-line statefile removal with full rewrite from in-memory hash: instead of reading the file, filtering one entry, and writing back, rewrite the entire file from the hash via bgp_rewrite_vrf_rd_statefile() and bgp_evpn_rewrite_vni_rd_statefile(). This is simpler and avoids format-parsing fragility. 4. Add 'used' flag to vrf_rd_state_entry and vni_rd_state_entry: set the flag when an entry is consumed during config replay so orphan detection is possible after config load. 5. Post-config orphan cleanup: register bgp_rd_state_cleanup_after_config on the bgp_config_end hook. After config is fully loaded, walk both hashes and remove entries whose 'used' flag is still false (VRFs/VNIs that were in the state file but no longer in frr.conf). Release their rd_id bits and rewrite the state files. 6. Proper hash cleanup at shutdown: free vrf_rd_state and vni_rd_state hashes in bgp_exit() to avoid memory leaks on termination. Signed-off-by: Soumya Roy <[email protected]>

Verify that RD state files are created correctly at startup, RD values remain stable across bgpd restarts, and orphan entries are cleaned from state files when VRFs are removed from config. Signed-off-by: Soumya Roy <[email protected]> Made-with: Cursor

frrbot bot added bgp bugfix tests Topotests, make check, etc labels Mar 19, 2026

soumyar-roy marked this pull request as draft March 19, 2026 23:38

github-actions bot added master size/XL labels Mar 19, 2026

greptile-apps bot reviewed Mar 19, 2026

View reviewed changes

bgpd/bgp_main.c Outdated Show resolved Hide resolved

bgpd/bgp_evpn.c Outdated Show resolved Hide resolved

bgpd/bgp_evpn.c Outdated Show resolved Hide resolved

bgpd/bgpd.c Outdated Show resolved Hide resolved

soumyar-roy force-pushed the soumya/cacherd branch 2 times, most recently from 5c4bfb8 to 06a7696 Compare March 20, 2026 18:08

mjstapp reviewed Mar 20, 2026

View reviewed changes

soumyar-roy force-pushed the soumya/cacherd branch 6 times, most recently from 9f7a481 to fa423e3 Compare March 23, 2026 19:40

Pdoijode added 2 commits March 24, 2026 10:17

Save rd_id to file and read from it

6254797

fixed compilation err

8595b92

soumyar-roy force-pushed the soumya/cacherd branch from fa423e3 to ee8a371 Compare March 24, 2026 17:18

soumyar-roy marked this pull request as ready for review March 24, 2026 21:55

ton31337 self-requested a review March 25, 2026 08:25

soumyar-roy added 2 commits March 25, 2026 11:17

soumyar-roy force-pushed the soumya/cacherd branch from ee8a371 to c88019b Compare March 25, 2026 18:18

Conversation

soumyar-roy commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem Statement

Why Alternative Approaches Fall Short

"Configure explicit RDs via orchestration"

"Use a different RD format"

Proposed Design

Overview

Scope of State

Lifecycle

Data Structures

File Format

Failure Modes

Orphan Cleanup

Files Modified

What This Does NOT Change

Testing

================================================================================ TEST : Final stability — "write memory" to persist re-added vrf1, restart FRR, verify RDs are identical. Expect: All RDs unchanged after restart. Final steady state.

Uh oh!

greptile-apps bot commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 3/5

Important Files Changed

Flowchart

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ton31337 commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mjstapp left a comment

Choose a reason for hiding this comment

Uh oh!

soumyar-roy commented Mar 23, 2026

Uh oh!

soumyar-roy commented Mar 24, 2026

Uh oh!

soumyar-roy commented Mar 24, 2026

Uh oh!

ton31337 commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

soumyar-roy commented Mar 24, 2026

Uh oh!

greptile-apps bot commented Mar 24, 2026

Uh oh!

sudharr86 commented Mar 24, 2026

Uh oh!

soumyar-roy commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

soumyar-roy commented Mar 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

soumyar-roy commented Mar 19, 2026 •

edited

Loading

================================================================================
TEST : Final stability — "write memory" to persist re-added vrf1,
restart FRR, verify RDs are identical.
Expect: All RDs unchanged after restart. Final steady state.

greptile-apps bot commented Mar 19, 2026 •

edited

Loading

ton31337 commented Mar 20, 2026 •

edited

Loading

ton31337 commented Mar 24, 2026 •

edited

Loading

soumyar-roy commented Mar 24, 2026 •

edited

Loading